Description

Part of #59.

Implementation Details

The refactoring is somewhat extensive - which makes the changes difficult to follow - but the changes can be summarised as follows:

• multiple caches are now handled independently of one another and have been moved out into TopologyCache
• function buildNodeToDatanodeMap has been introduced to cache a map of dataNode names to IPs: this enables O(1) lookup when finding co-located dataNodes for client pods
• program flow has been changed to hopefully make things more readable. The resolution flow consists of:
  • return topology from cache if possible, otherwise:
  • fetch dataNodes and prepare maps of their labels
  • build node-to-datanode map to use for O(1) colocated lookups
  • for each name, try the following in this order (enabling us to short-circuit on the first match):
    • attempt a direct dataNode lookup
    • fetch listeners, resolve them to their dataNodes and see if we have a match
    • fetch pods, find the co-located dataNode for the calling Pod
    • build and cache the topology
• a podInformer is used to watch the namespace for pod changes and to thereby reduce the chane of cache-misses

How to test

• Deploy hdfs-operator from this PR: chore: Added permission for topology-provider hdfs-operator#738 (has required permission to inspect listeners to access the version)
• Build and load the image from this PR: chore(hdfs-utils): Reference refactored topology provider docker-images#1360
• run the topology-provider integration test, specifying the custom image as loaded in the previous step

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Reviewer

• Code contains useful comments
• (Integration-)Test cases added
• Documentation added or updated
• Changelog updated

Acceptance

• Proper release label has been added